ML Course - Introduction to Machine Learning

Alexandre Bry

Definitions

Machine Learning (ML)

Machine Learning (ML)

Feeding data into a computer algorithm in order to learn patterns and make predictions in new and different situations.

ML Model

Computer object implementing a ML algorithm, trained on a set of data to perform a given task.

Categories of ML

Type of dataset

The four main categories of ML are based on the type of dataset used to train the model:

Supervised Unsupervised Semi-supervised Reinforcement
Input Data Data Data Environment
Ground-truth Yes No Partial No (reward)
Examples Classification, Regression Clustering Anomaly detection Game playing

Type of output

Another way to categorize ML models is based on the type of output they produce:

Category Description Example Outputs Example Use Cases
Classification Assign one (or multiple) label(s) chosen from a given list of classes to each element of the input. “Cat”, “Dog”, “Bird” Spam detection, Image recognition
Regression Assign one (or multiple) value(s) chosen from a continuous set of values. 3.5, 7.2, 15.8 Stock price prediction, Age estimation
Clustering Create categories by grouping together similar inputs. Cluster 1, Cluster 2 Customer segmentation, Image compression
Anomaly Detection Detect outliers in the dataset. Normal, Outlier Fraud detection, Fault detection
Generative Models Generate new data similar to the training data. Image, Text, Audio Image generation, Text completion
Ranking Arrange items in order of relevance or importance. Rank 1, Rank 2, Rank 3 Search engine, Recommendation system
Reinforcement Learning Learn a policy to maximize long-term rewards through interaction with an environment. Policy, Action sequence Game playing, Robotics control
Dimensionality Reduction Reduce the number of features while retaining meaningful information. 2D or 3D projection Visualization, Data compression

Dataset

Dataset - Definition

Dataset

A collection of data used to train, validate and test ML models.

Dataset - Example

Dataset example

sepal length sepal width petal length petal width species
0 4.8 3.1 1.6 0.2 0
1 6.5 2.8 4.6 1.5 1
2 5.4 3.9 1.7 0.4 0
... ... ... ... ... ...
147 7.7 3.0 6.1 2.3 2
148 7.7 3.8 6.7 2.2 2
149 4.6 3.2 1.4 0.2 0

150 rows × 5 columns

Iris dataset example

Content - Definitions

Instance (or sample)

An instance is one individual entry of the dataset (a row).

Feature (or attribute or variable)

A feature is a piece of information that the model uses to make predictions.

Label (or target or output or class)

A label is a piece of information that the model is trying to predict.

Feature vs. Label

Features and labels are simply different columns in the dataset with different roles.

Content - Example

Instances, features and labels

Feature 1 Feature 2 Feature 3 Feature 4 Label
Instance 0 4.8 3.1 1.6 0.2 0
Instance 1 6.5 2.8 4.6 1.5 1
Instance 2 5.4 3.9 1.7 0.4 0
... ... ... ... ... ...
Instance 147 7.7 3.0 6.1 2.3 2
Instance 148 7.7 3.8 6.7 2.2 2
Instance 149 4.6 3.2 1.4 0.2 0

150 rows × 5 columns

Instance, feature and label in the Iris dataset

Subset

Dataset subsets

A ML dataset is usually subdivided into three disjoint subsets, with distinctive role in the training process:

  • Training set: used during training to train the model,
  • Validation set: used during training to assess the generalization capability of the model, tune hyperparameters and prevent overfitting,
  • Test set: used after training to evaluate the performance of the model on new data it has not encountered before.

Overview of ML methods

Supervised Learning

Decision Trees - Definition

A tree-like structure used for both classification and regression

Simple decision tree on the Iris dataset

Decision Trees - Example

Decision boundaries of a decision tree on the Iris dataset

Random Forests - Definition

An ensemble method that combines multiple decision trees:

  • Train independently \(B\) trees using:
    • Bagging: each tree is fitted on a random subset of the training set
    • Feature bagging: each split in the decision tree (i.e. each node) is chosen among a subset of the features
  • Take a decision by aggregating individual decisions of each tree

Random Forests - Example

Decision boundaries of a random forest on the Iris dataset

Support Vector Machines (SVM) - Definition

Used for classification and regression, effective in high-dimensional spaces:

  • Separate the feature space using optimal hyperplanes
  • Features are mapped in a higher dimensional space to allow to fit non-linearly in the original feature space

SVM: map features to a higher dimensional space to be able to separate classes using a hyperplane[1]

Support Vector Machines (SVM) - Example

Decision boundaries of a SVM with the RBF kernel on the Iris dataset

Other methods

Other methods for supervised learning include:

Method Description
Linear Regression Predicts a continuous value with a linear model
Logistic Regression Predicts a binary value with a linear model
K-Nearest Neighbors (KNN) Non-parametric method for classification and regression
Boosting Ensemble method (like Random Forests) that combines weak learners to form a strong model
Naive Bayes Probabilistic classifier based on Bayes’ theorem

Unsupervised Learning

K-Means Clustering

A method for partitioning data into \(k\) clusters:

  • \(k\) must be chosen a priori
  • The principle is to start with random centroids and then iteratively:
    1. Classify points using the closest centroid
    2. Move each centroid to the real centroid of its class
  • Classical method with lots of variation

K-Means Clustering convergence process[2]

Hierarchical Clustering

Builds a hierarchy of clusters using either agglomerative or divisive methods:

  • Build a full hierarchy top-down (divisive) or bottom-up (agglomerative)
  • Create any number of clusters by cutting the tree

Hierarchical Clustering[3]

Density-Based Spatial Clustering of Applications with Noise (DBSCAN)

Clustering based on the density of data points:

  • Divides points in 4 categories: core points (in red below), directly reachable (yellow), reachable and outliers (blue)
  • Only two parameters: radius size (\(\epsilon\)) and number of neighbors to be core (\(min_{pts}\))

DBSCAN Illustration[4]

DBSCAN Result[5]

Other methods

Other methods for unsupervised learning include:

Method Description
Gaussian Mixture Models (GMM) Probabilistic clustering assuming data is generated from multiple Gaussian distributions
Autoencoders Neural networks that learn efficient representations of data in an unsupervised manner
Self-Organizing Maps (SOM) Neural network-based method for clustering and visualization

Dimensionality Reduction

PCA - Definition

PCA (Principal Component Analysis) is a dimensionality reduction technique to project data into lower dimensions:

  • Project data into a space of lower dimension
  • Keep as much variance (so as much information) as possible

PCA Illustration[6]

PCA - Example

PCA on the Iris dataset

t-SNE - Definition

t-SNE (t-Distributed Stochastic Neighbor Embedding) is a nonlinear dimensionality reduction technique primarily used for visualization of high-dimensional data

t-SNE Result on MNIST dataset[7]

t-SNE - Example

t-SNE on the Iris dataset

Neural Networks

Definitions

Neural Network (NN)

Neural Network (NN)

Subtype of ML model inspired from brains. Composed of several interconnected layers of nodes capable of processing and passing information.

Deep Learning (DL)

Subcategory of Machine Learning. Consists in using large NN models (i.e. with a high number of layers) to solve complex problems.

Common representation of a neural network

Structure

Structure of a NN

  • Neuron: Base element. Takes multiple inputs, sums them with weights and passes the result as output.
  • Layer: Set of similar neurons taking different inputs and/or having different weights.
  • Structure: Ensemble of layers composing the model, including their types, sizes, succession and parameters.

Types of Layers

Dense (or Fully Connected) - Diagram

The most basic layer, in which each output is a linear combination of each input (before the activation layer).

Fully connected layer

Dense (or Fully Connected) - Math

The output of a dense layer is computed as follows:

\[ \require{colortbl} \underbrace{ \left[ \begin{array}{c} \rowcolor{#ffd077} x_1 & x_2 & x_3 & x_4 \end{array} \right] }_{\text{Input}} \cdot \underbrace{ \left[ \begin{array}{c} \columncolor{#ffd077} w_{11} & w_{12} & w_{13} \\ w_{21} & w_{22} & w_{23} \\ w_{31} & w_{32} & w_{33} \\ w_{41} & w_{42} & w_{43} \end{array} \right] }_{\text{Weights}} = \underbrace{ \begin{bmatrix} \columncolor{#ffd077} y_1 & y_2 & y_3 \end{bmatrix} }_{\text{Output}} \]

with \(y_1 = x_1 w_{11} + x_2 w_{21} + x_3 w_{31} + x_4 w_{41}\)

Convolutional - Diagram

A layer combining geographically close features, used a lot to process rasters.

Convolutional layer - real input shape

Convolutional layer - easier to visualise

Convolutional - Math

The output of a convolutional layer is computed as follows:

\[ \require{colortbl} \underbrace{ \left[ \begin{array}{c} \cellcolor{#ffd077}x_{11} & \cellcolor{#ffd077}x_{12} & x_{13} \\ \cellcolor{#ffd077}x_{21} & \cellcolor{#ffd077}x_{22} & x_{23} \\ x_{31} & x_{32} & x_{33} \end{array} \right] }_{\text{Input}} \ast \underbrace{ \left[ \begin{array}{c} \cellcolor{#ffd077}w_{11} & \cellcolor{#ffd077}w_{12} \\ \cellcolor{#ffd077}w_{21} & \cellcolor{#ffd077}w_{22} \end{array} \right] }_{\text{Kernel/Weights}} = \underbrace{ \begin{bmatrix} \cellcolor{#ffd077}y_{11} & y_{12} \\ y_{21} & y_{22} \end{bmatrix} }_{\text{Output}} \]

with \(y_{11} = x_{11} w_{11} + x_{12} w_{12} + x_{21} w_{21} + x_{22} w_{22}\)

Recurrent

Type of layers designed to process sequential data such as text, time series data, speech or audio. Works by combining input data and the state of the previous time step.

The two main variants of recurrent layers are:

Long Short-Term Memory (LSTM)[8]

Gated Recurrent Unit (GRU)[9]

Nowadays, transformer architectures are however preferred to process sequential data.

Pooling

A type of layers used to reduce the number of features by merging multiple features into one. There are multiple kinds of pooling layers, the most simple ones being Maximum Pooling and Average Pooling.

Max Pooling Example[10]

Other Types

Type Description
Residual Skips some layers to improve training
Attention Focuses on specific parts of the input data
Embedding Transforms discrete data into continuous vectors
Dropout Randomly drop out some of the nodes during training to reduce overfitting
Batch Normalization Normalizes the input of each layer across the batch to improve training stability and speed
Layer Normalization Normalizes the input of each layer across the features to improve training stability and speed
Embedding Transforms discrete input data into continuous vectors with lower-dimensional space
Flatten Convert multi-dimensional data into 1D data that can be fed into fully connected layers

Architecture and Weights

Both are Crucial

A NN is defined by both its architecture and its weights

  • Architecture: The structure of a NN, including the type and number of layers, their size and succession.
  • Weights: The values used to sum the inputs of each node in the network. They are learned during the training process.

Both are Crucial - Without Weights

Architecture without weights is useless

Both are Crucial - Without Architecture

Two architectures with the same weights but doing very different things:

A fully connected architecture

A convolutional architecture

Both are Crucial - Conclusion

Both the architecture and the weights are needed

When importing a model, you need to import both the architecture and the weights. The process varies depending on the library used.

This also means that the having access to the architecture is not enough to reproduce the model.

Training Process

Objective

What do we want?

We want a model that performs well on a given task. To achieve this, we need to:

  1. Select the right architecture for the model (not covered in these slides)
  2. Find the right weights for the model

Right Weights

What are the right weights?

The right weights are the ones that allow the model to perform well on the task. More precisely:

  • Performing well means minimizing a loss function
  • On the task means on the dataset that we have for the task

Loss function

Function that evaluates the performance of a model. In supervised learning, it compares the predictions of the model to the ground-truth.

General Idea

General idea

Since there is a huge number of possible combinations of weights, we need to search for the right ones. This process is called training. It is iterative and consists of the repetition of three steps:

  1. Forward pass: compute the output of the model on the training data
  2. Backward pass: compute the gradient of the loss function with respect to the weights
  3. Optimization: update the weights using the gradient

The process is repeated until the model performs well on a validation set.

Animation - What Happens

Animation of the training process

Animation - What We Really Know

Animation of the training process without the known loss function

Animation - Sometimes It Works Well

It works well

It works well

It works

Animation - Sometimes It Doesn’t Work

It doesn’t work

It doesn’t work

It doesn’t work

Project Pipeline

Overview

  1. Data acquisition
  2. Data preprocessing
  3. Model selection
  4. Model optimization/training
  5. Model evaluation
  6. Go back to 3 if not satisfied
  7. Final model training

Data Preprocessing

Different Issues

Multiple sources of issues and steps to perform:

  1. Handle different formats
  2. Remove outliers (mostly for raw data)
  3. (Optionally) extract features
  4. Handle missing data
  5. Normalize

Why Normalization?

No prior knowledge

A priori all features have the same importance, so none of them should have an advantage. Therefore, having features with larger values than others would be detrimental.

Usually, all features are individually normalized over the whole dataset, to obtain a distribution with an average of 0 and a standard deviation of 1:

\[ \begin{align*} \hat{X} & = \sum\limits_{j=0}^n X_j \\ \sigma_X & = \sum\limits_{j=0}^n (X_j - \hat{X})^2 \\ \forall k \in [0, \cdots, n ], X_k & = \frac{X_k - \hat{X}}{\sigma_X} \end{align*} \]

Model Selection

  • Type of model (ML, NN, DL, …)
  • Complexity:
    • Number of features
    • Type of output
    • Size of the layers (for NN)
    • Number of layers (for NN)
    • …

Model Optimization / Training

  • Loss selection: depends on the task, the objectives, the specific issues to solve
  • Training process selection (lots of different tweaks and improvements can be implemented in NN training)
  • Hyperparameter tuning, by repeatedly:
    • Selecting one or multiple configurations of hyperparameters
    • Training the model one or multiple times
    • Determining the best hyperparameters

Model Evaluation

Criteria - Classification

Most common evaluation criteria for classification tasks:

Name Use case Formula
Accuracy Balanced datasets \(\frac{TP + TN}{TP + TN + FP + FN}\)
Precision False positives are costly \(\frac{TP}{TP + FP}\)
Recall False negatives are costly \(\frac{TP}{TP + FN}\)
F1-Score Unbalanced class distribution \(\frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}\)
… … …

Criteria - Regression

Most common evaluation criteria for regression tasks:

Name Use case Formula
Mean Absolute Error (MAE) Robust to outliers \(\frac{1}{n} \sum\limits_{i=1}^n |y_i - \hat{y}_i|\)
Mean Square Error (MSE) Sensitive to large errors \(\frac{1}{n} \sum\limits_{i=1}^n (y_i - \hat{y}_i)^2\)
… … …

Cross-validation

Cross-validation

Method to estimate the real performance of the model:

  1. Split the dataset in multiple parts (usually 5)
  2. Train and evaluate the model for different combinations of these parts (usually 5)

Cross-validation[11]

Final Model Training

Once the data is preprocessed, the model is selected, the hyperparameters chosen and optimized, the final model can be trained (potentially multiple times to keep the best one).

Challenges

Data

Quality

Quality of the data is obviously crucial to train well-performing models. Quality encompasses multiple aspects:

  • Raw data quality: the input data must possess high enough details for the task to be even achievable. Be careful however as more features imply larger models which are longer and harder to train.
  • Annotations quality: the annotations must be as precise and correct as possible in the context of the task at hand. Every blunder or outlier in a supervised dataset will slow down training and might result in unexpected behaviors of the trained model.

Diversity

Diversity is the most important aspect of a dataset because ML models are great at generalizing but bad at guessing in new scenarios. There are different aspects to diversity to keep in mind:

  • A well-defined task is crucial to identify all the various cases that we want our model to handle. Being as exhaustive as possible when selecting the training instances will accelerate training and improve the model by a lot
  • Balancing the dataset can also improve the training. When training on imbalanced datasets (i.e. when some cases are much more represented than others), the model will focus on the most represented situations as it will be the easiest and quickest way to get better results. There are ways of correcting this phenomenon, but it is always better to avoid it if possible when building the dataset.

Biases and Fairness

Biased

Refers to a model which always makes the same kind of wrong predictions in similar cases.

In practice, a model trained on biased data will most of the time repeat the biased results. This can have major consequences and shouldn’t be underestimated: even a cold-hearted ML algorithm is not objective if it wasn’t trained on objectively chosen and annotated data.

There exist model architectures, training and evaluation methods to try to prevent and detect biases. They can sometimes allow to build unbiased models using biased data. But this adds complexity to the training process and doesn’t always work.

Underfitting and Overfitting

Definitions

Underfitting

When a model is too simple to properly extract information from a complex task. Can also be explained by key information missing in the input features.

Overfitting

When a model is too complex to properly generalize to new data. Happens often when a NN is trained too long on a dataset that is not diverse enough and learns the noise in the data.

Illustrations

Underfitting and overfitting on a regression task[12]

Underfitting and overfitting on a classification task[12]

Solutions

Lever In case of underfitting In case of overfitting
Complexity Increase Reduce
Number of features Increase Reduce
Regularization Reduce Increase
Training time Increase Reduce

General strategies:

  • Cross-validation to identify problems
  • Grid search/random search to tune hyperparameters and balance between underfitting and overfitting
  • Ensemble methods to reduce overfitting by using many smaller models instead of one big

Interpretable and Explainable

Definitions

Interpretable

Qualifies a ML model which decision-making process is straightforward and transparent, making it directly understandable by humans. This requires to restrict the model complexity.

Explainable

Qualifies a ML model which decision-making process can be partly interpreted afterwards using post hoc interpretation techniques. These techniques are often used on models which are too complex to be interpreted.

Python Libraries

Data Manipulation

  • NumPy
    • Fast numerical operations
    • Matrices with any number of dimensions (called arrays)
    • Lots of convenient operators on arrays
  • Pandas
    • Can store any type of data
    • 1D or 2D tables (called DataFrames)
    • Lots of convenient operators on DataFrames

ML

  • SciPy
    • Scientific and technical computing based on NumPy
    • Lots of tools for optimization, integration, interpolation, etc.
  • scikit-learn
    • Simple and efficient tools for data mining and data analysis
    • Built on NumPy, SciPy and matplotlib

Visualization

  • Matplotlib
    • 2D plotting library (3D also possible)
    • Can create plots, histograms, power spectra, bar charts, error charts, scatterplots, etc.
    • Gallery
  • Plotly
    • Similar to Matplotlib but with interactive plots
    • Gallery
  • Seaborn
    • Data visualization library based on Matplotlib and Pandas
    • Very powerful for statistical data visualization
    • Gallery

Resources

References

Introduction to Machine Learning

[1] Vector: Zirguezi Original: Alisneaky. “SVM kernel trick: Map features to a higher dimensional space to be able to separate classes using a hyperplane.” Available at: https://commons.wikimedia.org/w/index.php?curid=47868867.
[2] Chire. “K-means clustering convergence process.” Available at: https://commons.wikimedia.org/w/index.php?curid=59409335.
[3] derivative work: Mhbrugman Stathis Sideris. “Hierarchical clustering.” Available at: https://commons.wikimedia.org/w/index.php?curid=7344806.
[4] Chire. “DBSCAN illustration.” Available at: https://commons.wikimedia.org/w/index.php?curid=17045963.
[5] Chire. “DBSCAN result.” Available at: https://commons.wikimedia.org/w/index.php?curid=17085332.
[6] Nicoguaro. “PCA illustration.” Available at: https://commons.wikimedia.org/w/index.php?curid=46871195.
[7] Kyle McDonald. “T-SNE result on MNIST dataset.” Available at: https://commons.wikimedia.org/w/index.php?curid=115726949.
[8] fdeloche. “Long short-term memory (LSTM).” Available at: https://commons.wikimedia.org/w/index.php?curid=60149410.
[9] fdeloche. “Gated recurrent unit (GRU).” Available at: https://commons.wikimedia.org/w/index.php?curid=60466441.
[10] Daniel Voigt Godoy. “Max pooling example.” Available at: https://github.com/dvgodoy/dl-visuals/.
[11] Gufosowa. “Cross-validation.” Available at: https://commons.wikimedia.org/w/index.php?curid=82298768.
[12] GeeksforGeeks. “Underfitting and overfitting in machine learning.” Available at: https://www.geeksforgeeks.org/underfitting-and-overfitting-in-machine-learning/.